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Abstract 

Given a probability distribution Px, what is the minimum amount 
of bits needed to store a value x sampled according to Px, such that x 
can later be recovered (except with some small probability e)? Or, what 
is the maximum amount of uniform randomness that can be extracted 
from x7 Answering these and similar information-theoretic questions typ- 
ically boils down to computing so-called smooth entropies. In this paper, 
we derive explicit and almost tight bounds on the smooth entropies of 
n-fold product distributions Px- 



1 Introduction 

1.1 Smooth min- and max-entropy 

Smooth min- and max-entropy has been introduced in |RW04I IRWOSj as a gen- 
eralization of Shannon entropy. Similarly to Shannon entropy, smooth min- 
and max-entropy can be used to analyze information-processing tasks such as 
data compression. However, in contrast to Shannon entropy, which usually 
only makes sense in an asymptotic setting (where an underlying random ex- 
periment is repeated many times), smooth entropies can also be used in the 
non-asymptotic case. 

We start by quickly reviewing the relevant definitions. For the following, 
let X and Y be random variables with range X and 3^, respectively, and joint 
distribution Pxy- Moreover, for e > 0, let B^{Pxy) be the e-ball of nonnegative 
functions around Pxy, i-e., the set of functions Qxy : X y.y ^ M+ such that 
\\PxY — Qxy\\i < £, where || • ||i denotes the Li-norm. 

Definition 1. The e-smooth max-entropy of X given Y is^ 

■ffmax(^l^) — ^ min„ maxlog|suppgxy(-,2;)| 

QxYeB'iPxY) yey 
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where supp Qjcy (•, y) denotes the support of the function QxY{',y) ■ x 
QxY{x,y). 

The e-smooth min-entropy of X given Y is 

Pyiy) 

H^^^{X\Y) := max min minlog- 



JXYk 



^eB^{PxY)y&iiuppPY xex QxY{x,y) 
where Py denotes the marginal distribution of Pxy ■ 

The following two statements proven in |RW05| imply that the smooth min- 
and max-entropies have a (non-asymptotic) operational interpretation. For ex- 
ample, the smooth max-entropy characterizes data compression. More precisely, 
for {x,y) chosen according to Pxy, it quantifies the minimum space needed to 
store X such that, with the help of y, the value x can later be retrieved (except 
with probability at most e). 

Proposition 1. Let (,^^^^{X\Y) he the minimum number I such that 

Pr \d{e{x),y) ^ x\ < e 

{x,y)^PxY 

for some encoding function e : X {0, 1}^ and some decoding function d : 
{0, 1}^ X 3^ ^ X. Then, for any < e' < 

Hl.AX\Y) < eLciX\Y) < Hi^,{X\Y) + log(l/(£ - e')) + 1 . 

Similarly, the smooth min-entropy characterizes randomness extraction. That 
is, for (x, y) chosen according to Pxy, it corresponds to the maximum number of 
bits that can be computed from x such that these bits are uniformly distributed 
and independent of y (except with probability e). 

Proposition 2. Let £^^^{X\Y) be the maximum number £ such that 

1 II II 
■^\\Ph{x)Y - Pu^ Py\\^ < e 

for some extraction function h : X ^ {0, 1}^, where Pjj is the uniform distri- 
bution on {0, 1}^. Then, for any {)<£'<£, 

H<,{X\Y) - 21og(l/(£ - £')) < ^Lt{X\Y) < H-^.JX\Y) . 

While, by the above propositions, smooth entropies are directly related 
to data compression and randomness extraction, they are also useful for the 
characterization of a variety of other tasks in communication theory and cryp- 
tography |RW05I IRWWOBj . Moreover, they can be generalized to quantum 
states |R.en05| . 



2 



1.2 Contributions of this paper 

In this paper, we are concerned with the exphcit computation of smooth en- 
tropies for the case of a finite number of independently repeated experiments. 
More precisely, we derive the following bounds on the smooth min- and max- 
entropies H^-^^{X"\Y") and H^^^^{X"\Y") of an n-fold product distribution 
-Px"y- := PxiYi X • • • X Fx„y„- 

Theorem 1. Let Px"Y" ■= PxiYi ■ ■ ■ Px„y„ be a probability distribution over X^x 
3^". For any6>0 

i7^,,(X"|r")<i/(X"|r") + n<5, 

i/^ijx"|r")>i/(x"|r")-n5 , 

where s = 2 ^log^d^i+s) , 

Let (x,y) be chosen according to the n-fold product distribution Px^Y" = 
PxiYi ■ ■ ■ Px„Y„- The well known asymptotic equipartition property states that 

hm (Pr[P^„|^„(x,y)e2-^(^"l^")±"^])=l 

n — »oo x,y 

for every 5 > 0. The main step in order to prove Theorem ^ is to give the 
following quantitative bound on this convergence. 

Theorem 2. Let Px^y^ '■= PxiYi ■ ■ ■ Px„y„ be a probability distribution over X^^x 
. Then, for any 5 G [0, log(| A"!)] and (x, y) chosen according to Px^y^j 

Pr[-log(Px"|y"(x,y)) > H{X^\Y^) + n5\ <e, 

and, similarly, 

Pr[-log(P;,„|y„(x,y)) <i/(X"|y")-n(5] <£ , 

x,y'- 

where e = 2 ^log^d^i+s) , 

We prove the theorems in Section [3 In Section Owe show that both these 
theorems are almost tight (cf. Theorems O and 0J . 

1.3 Related work and proof technique 

A bound as in Theorem |21 can be obtained in simpler ways than the one we use. 
However, we only know of simpler arguments which yield quantitatively weaker 
bounds. 

We sketch two such arguments. The first argument (which only seems to 
work in case the distributions PxiYi are identical) goes as follows: let Qx|y be the 
frequency distribution (i.e., the type) of the pair (x, y) — {{xi, yi), . . . , (a;„, ?/„)). 
It is well known that D{Q^\y\\Px\Y) is small with high probability, and an 
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explicit bound can be given |CT91I Theorem 12.2.1]. The Csiszar-Kullback- 
Pinsker inequahty |CT91I Lemma 12.6.2] then shows that ||Qx|y — -Px|f||i is 
small with high probability, which in turn bounds \H{Q^\y) — H{Px\y)\ by the 
Fano inequality (more concretely |CT91[ Theorem 16.3.2]), and a bound on 
|i/((5x|y) ~ H{Px\y)\ is exactly what we want. This argument shows that the 
probabilities in Theorem |21 are at most 

The major problem with this bound is that it is only useful if n 3> but ad- 
ditionally the term log^(| -Yj/J) is not tight: it can be strengthened to log^dA"]) 
as our proof shows (which is interesting if 5 <C j^)- 

A different argument is used in |ILL89| . There, the Hoeffding bound is 
applied on the sum of the independent random variables log(l/Px|V'(^j7 ^))- 
Unfortunately, the Hoeffding bound can only be applied if the random variables 
have a bounded range, and thus one ignores occurrences where this random 
variable is very large (which happens with some small probability). Hence, this 
technique only gives a bound on the second probability in Theorem |21 and this 
bound is 

2""-^( io«^(f^i/^) ). 

As above, this is not tight in case 6 ^ yyy. 

Instead of using Hocffding's bound we directly use Chernoff's argument 
| Che52| which states that an upper bound on inft e^°-^AI{t) for every a suffices 
for our purpose, where M(t) is the moment generating function of the random 
variable log(l/Px|y(^: ^))- In order to make the presentation simpler we do 
not use Chernoff's theorems explicitly, but instead give the complete proof. 

2 Smooth min- and max-entropy of products 

2.1 Typical sequences and their probabilities 

Lemma 1. Let Pxy be a probability distribution on X x y. Then, for any 

t e M with \t\ < i,g(|^|^3) , 

\og(E^,y[PxiY{x,y)-']) <tH{X\Y) + ^tHog\\X\+3) , 

where the expectation is taken over pairs {x,y) chosen according to PxY ■ 
Proof. For any t G M, let rt be the function on the open interval (0, oo) defined 

by 

rt(z) :=z*-<ln(2)-l . (1) 
We will use several properties of this function proven in Appendix 
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For any x G X and y E y, let p^.y Px\y{x, y)- If Px,y > then 

P'^l - + 1 ^ ^*(^ + 3) + ^ln(^rb) + 1 ' 

where the inequaUty holds because rt is monotonically increasing on the interval 

fl,oo) (Lemma 01 and — p^^f^^ ^ > 1. Because — 1-3 G [4, oo) and 

because rt is concave on this interval (Lemma |S1 which can be applied because 
is [— ^, ^]), Jensen's inequality leads to 

r , , ,n . , . ,n 



E.,,[p-;] <E,,,[r,(^+3)J +iE,,,[ln(^ 

< n (e,^, [-1- + 3] ) + t ln(2) E,,, [log(^) 



1 



Because E.,,[^] = ^^^^ P^^(x, = l^-] and E,,,[log(^)] = i/(X|y), 

we obtain 



E.,,[p-*,] <n{\X\+2,)+t\n{2)H{X\Y) + l . 
Furthermore, because log(a) < j^^{cl — 1), 

log(E,,,[p-*j) < ^^n{\X\+i)+tH{X\Y) . 

Finally, together with Lemma El since \t\ < logd^'i+a) ■> '^^ conclude 

log(E,,,[p-y) < {j^^-\y\oi{\X\+?,) + tH(X\Y) . 

The assertion follows because j^^j) ~ ^ ^ \- 

Lemma 2. Let Pxy be a probability distribution and let 7 be the function on 
X X y defined by 

7(x, y) := - \og{Px\Y{^, y)) - H{X\Y) . 
Then, for any t G R with \t\ < i^^^j^ppgy; 

E,,J2*'^("'«)] < 2^*'i°s'(l'^l+3) . 
Proof. The assertion follows directly from Lemma ^ that is, 
E.,,[2*^(--^)] = 2-*^(^l^)E.,,[P;,|y(x,y)-*] 

^ 2~tHiX\Y) , 2*J^(^l>') + 5*'l°g'(l'^l+3) . □ 

Proof (of Theorem\^. Let x — (xi, . . . ,a;„), y = (?/i, . . . and let 7^ be the 
function defined in Lemma [21 for the probability distribution PxtYi- Then 

n 

J2^^{x,^y^) = -log(Px"|y"(x,y)) -i/(X"|y") . (2) 
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Using Markov's inequality, for any i > 0, 



n 

Pr[V7,(x„y,) > nS] = Pr [2*^"=^ > 2*"*] < 

x,y L ' ^ J x,y 



Moreover, because the pairs (xi,yi) are chosen independently, 



(3) 



Ex,y[j|2*''*(^"2'') 

n 

J|E,,,y,[2*'^'("'*)] < (2^t'i°g'(l^l+3))" 



where the inequahty follows from Lemma 13 for any |t| < logdA'i+s) ■ Combining 
this with Q gives 



n 

x,y — ' 

2—1 



> n8 



With t := iog:j(|^|^3) (note that i < iog(|;^|^3) because (5 < logdA"!)), we con- 
clude 



n 



< 2 2 1og^(|^|+3) 



The first inequality of the theorem then follows from (PJ ■ 
Similarly, if i < 0, 



= Pr 



Pr V'7i(xi,yi) < -n6 

x.yL-"^ — ' 
z— 1 

and thus 

n 

Pr y'7»(a;i,yi) < -nS 

x,yL'' — ' 

i— 1 

The second inequality follows with t := 



< 



Ex,y[2*S?=l'^'(^"2/.)] 



2 — tn(5 



^ 25"*^ l°g^(l'^l+3)+*"'5 



loe^\X\+3)- 



□ 



2.2 Asymptotic equality of smooth entropy and Shannon 
entropy 

Proof ( of Theorem QJj . We first prove the bound on the max-entropy ff^^x I ) ■ 
For any y S iV" with Pyn (y) > 0, let Xy be the set of all n-tuples x g X" such 
that 

-log(Px"|y"(x,y)) < +n5 . 

Furthermore, let P^nyn be the nonnegative function on X" x J^" defined by 



Px„y„(x,y) 



Px'.i'"(x,y) ifxeA-y 
otherwise. 



(4) 
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We can assume without loss of generahty that 6 < logdA"]) (otherwise, the 
statement is trivial). Hence, by the first inequality of Theorem |21 Prx.y[x <^ 
Xy] < £, which implies 

||Px"F" -Px-y-lli <£• (5) 
For any fixed y := (yi, . . . , y„) G 3^" with Py^ (y) > 0, 

1> J2 ^x,|n(xl,yl)•••Px„|y„(^n,2;n)>|A^y|2-^(^"l^")-"^ 

where the second inequality follows from the definition of the set Xy. Conse- 
quently, we have \Xy\ < 2^(^"l^")+"''. Moreover, by the definition of Py„-p„, 
the support of the function x i— > P^„yn(x, y) is contained in Xy. By the defi- 
nition of max-entropy and jSJ 

H^^^{X-\Y^^) < maxlog|suppP^„y„(.,y)| <i/(X"|y") + n,5. 

To prove the bound on the min-entropy H^^^^Px^Y^lPy^), let Xy, for any 
y e 3^" with Pyn (y) > 0, be the set of n-tuples x G X" such that 

- log(Px„|y„ (x, y)) > H{X^\Y^) - n6 , 

and let again Pjf„y„ be defined by Q). By the second inequality of Theorem El 
Prx,y[x ^ Xy] < e, which gives 

||Px.y. -P^„y„|| <e . 

Thus, by the definition of min-entropy, 

Py„(y) 



i75,i„(X"|y") > min minlog- 



where the second inequality follows from the definition of the set Xy. □ 



3 On the Tightness of the Bounds 

In this section, we show that Theorems ^ and |5| are almost tight. For this, we 
construct one particular family of distributions for which these theorems cannot 
be strengthened much. 

Consider the set A" = {0, ... , \X\ - 1}, where \X\ > 3. We set 



otherwise. 
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An explicit calculation gives 



H{X)^l + -\og{\X\^l). 



(7) 



For a tuple x := (xi, . . . , a;„) let z(x) := |{i | Xi = 0}| be the number of zeros 
in X. Then, 

(- log(Px" (x))) - nH{X) - z(x) + (n - z(x))(l + logdA-j - 1)) - nH{X) 

^(|-z(x))log(|A'|-l). (8) 

Further, we note that 2:(x) is binomially distributed with p = i. 

We first show that Theorem |21 is almost tight. For this we use a lower 
bound on a partial sum over binomial coefficients which is given in Appendix 151 
(Lemma El- 

Theorem 3. For any X with \X\ > 3 there exists a distribution Px over X 
such that for n > f 2, < 5 < ^ ^fid fgj- x chosen according to Px" '-^ 

Pr[-log(Px"(x)) > n{H{X)+5)] > 
and, similarly, 



iog^(|;v|-i) 



Pr[-log(Px4x)) < n{H{X) - d)] > —2 



log^(IA-l-l) 



Proof. Let Px be the distribution defined by ©. We prove the second bound 
(the proof of the first bound is symmetric). From ^ we get 



Pr[-log(Px..(x)) < n{H{X) - S)] = Pr[(^- - z(x)j logdA"] - f ) < -nS 



Pr 

X 

Pr 



n nS 

^(x) < — ; 

2 ' '- \og{\X\ - 1) 



^(x) + 



nS 



2 logdA-l - 1) 



Using LemmaElwith s :— r iog(|'^|_i) 1 (it is easy to check that the requirements 
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of Lemma are satisfied) we get 
nS 



Pr 



<x) > - + 



2 logdA-l-l) 



> 



> 



2V^ 



> 



> 



21og(|A'|- 


if 


y/nS 




21og(|A'|- 


if 


VnS 




21og(|A'|- 


if 






leiogd-Yi- 


-1) 



e iog':(|;t|-i) 



2 log^d^i-i) 



(9) 



Fix now S* — . We consider the cases S > S* and S < S* separately. 

First, in case S > S* equation © implies 



Pr 



n nS 



> 2 l°i5^(l^|-l) 

64 



(10) 



On the other hand, ii 6 < 6* we use (|10|l to get 



Pr 



, , n 71(5 

^«^2 + kiPi-T) 



> Pr 



, , It 

zix) >- + 



nS* 



2 logdA-l - 1) 



1 3 1 
> — 2"^ > . 

- 64 110 



Since 1 > 2 '"s^d-^i-i) for all (5 > this finishes the proof. 
We now prove that Theorem ^ is almost tight. 



□ 



Theorem 4. For any X with \X\ > 3 there exists a distribution Px over X 
such that for n>l2Q0 and Q < 5 < '"^^ig^)"^^ 



1 

n 
1 



iJ^,,(X") > H{X) + ^ 
-Hl,^{X-) < H{X) - 5 



where e = -51772 "^^S^^w^) 

oc5U 



Proof. Again, let Px be the distribution defined by ©. First, let S C X^ be 
the set of values x whose probability is at most Px"(x) < 2~"(^(-^)+2'^+i"/"), 
and T C A"" be the set of values whose probability is at least Px"(x) > 
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2-niHix)-2S-io/n) p^^, ^^^^1 S and T, according to Theorem |3| (one can eas- 
ily check that Theorem O can be appHed for these parameters), the probabihty 
that X is in the set is at least 

I 12„(2i + 10/„)2 I 12,.(4i^+40V„+IIM) 

2 log^d^i-i) — 2 log^(i^i-i) 

110 110 

- 2 i°g^(i-¥i-i) . 2 log^d^i-i) 

110 

1 48Tij^ log(|;V|-l) + l 

> 2 i°g'-^(i-¥i-i) . 2 i°g^(i^i-i) 

- 110 

> 2 '"s^d-^-i-D = 2e. (11) 

- 440 ^ 

We now prove the bound on H^^^^. If Q G B^{Px") we have 

5] 0(X) > Px.(x) - ^ |PX"(X) - g(x)| > £, 

xGtS x€itS x(^tS 

where the last inequality follows because (|ll|l shows that X]xg5 ^^"^ (■"■) 
least 2e, while X^xgS l-f-x:"(x) ~ Qi^)\ is at most — QUi < £• 

We can assume that for all x e 5 we have Q(x) < 2-"(^(^)+2'5+io/«) (oth- 
erwise we find a function Q(x) which is closer to Px", has the same support, 
and satisfies this) and thus this means that 

Isuppgi >e2"(^(^)+2*'+io/"). 



I.e., 



Since 



log(| suppgi) > n{H{X) + 26+ lO/n) - log(l/£). 



log(l/e) = — 5 + log(880) <nS+ 10 



we get that 



-f^max = min log I supp Qx I 

> n{H{X) + 25+ 10/n) - ?i(5 - 10 = n{H{X) + 5) . 

We now come to the bound on the min-entropy H^-^:^^-^{X^) . If Q G B^{Px^) 
then e > ^^g^ |Px"(x) — Q(x)| > J2xeT -^x^i^) — Q{^) (where T is as defined 
above), and together with (|ll|l this implies 

^ g(x) >J2Px<^)-e>J2 IpxA^), 

xGT xGT xGT 

which implies that for some x e T we have Q(x) > 2-"(^(^)-'5). We therefore 
get 
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A Properties of the function rt 

This section lists some properties of the functions rt defined by i.e., 

rt{z) = z* - iln(z) - 1 . 
These properties are used in Section [21 

Lemma 3. For any i G R, the function rt is monotonically increasing on the 
interval [1, oo). 

Proof. The first derivative of rt is given by 

The assertion follows because the term on the right hand side is nonnegative for 
any z £ [1, oo). □ 

Lemma 4. For any i e K and z G (0, oo), 

rt{z)<r\t\{z + \) . 

Proof. Observe first that rt{z) — r_t(i). It thus suffices to show that the 
statement holds for t > 0. If 2; > 1, the assertion follows directly from Lemma^l 
For the case where i > and z < 1, let v := th\{\/z). Then ft(i) = e" — v — 1 
and rt{z) — e^^ + v — 1. Because w > 0, we have e" — e^^ > 2t;, which implies 
ft{z) < ft{\)- The assertion then follows again from Lemma|31 □ 

Lemma 5. For any t G [^^j^Ij function rt is concave on the interval 
[4,00]. 

Proof. We show that ■^rt{z) < for any z > 4. Because ■^rt{z) = t{t — 
l)z*^~'^ + -ij, this is equivalent to t{l — t)z^ > t. It thus suffices to verify that 



z > 



1 -t 



for any z > 4. If we substitute s := Y~i right hand side of the above 
expression is s^^ (where s G [|,2]), whose derivate can be easily seen to be 
non-negative (we use ln(s) < s — 1), and thus takes its maximum at s = 2, in 
which case it equals to 4. □ 

Lemma 6. For any z e [1, 00) and t G h 15^, i^^] 
rt{z) < (l-ln(2)) \og'{z)t^ . 
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Proof. Let V := iln(z). Then 

^ = ^ = .2 1^ (^) ■ (12) 

We first show that the term on the right hand side is monotonically increasing 
in V, that is, 



dv 

We multiply the last inequality with w^e""/^ on both sides and see that it is 
equivalent to 

which holds because, for any t; e M, 

W2 _ p-vjl 



ov/2 _j_ i;/2 



tanh(f)| < 



Hence, in order to find an upper bound on (|12f) . it is sufficient to evaluate 
the right hand side of \12\ for the maximum value of v. By assumption, we 
have V < ln(2), i.e., 

^--^In2(^)< (l-ln(2)) log2(z) , 
which concludes the proof. □ 

B Partial Sums over Binomial Coefficients 

Let 

B,ik\n):= (13) 

be the probability of obtaining k successes from n independent Bernoulli trials. 
We will also use the binary KuUback-Leibler distance D{q\\p), which is defined 
for arbitrary p,q (z [0, 1] by 

D{q\\p) q log(^) + (1 - g) log(^) • (14) 

Lemma 7. For p>^,£>0, p + e<l 

2 



D{p + e\\p) < 



e 



21n(2Ml-p)- 
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Proof. Define the function /p(e) '■— D{p + e\\p). Taylor's Theorem states that 
there exists a. 6 G [0, e] such that 



D{p + s\\p) = /,(£) = /p(0) + /;(0)e + /;'(0)- + f;\6)j. (15) 

ExpUcit calculation yields /p(0) = /^(O) = and /p (0) = in(2)p\i-p) • 
get 



(p + £)2(p + e- l)21n(2)' 
which is positive for e > 0. Together with (|15|) this gives the lemma. □ 
Proposition 3 (Stirling's Approximation). For any n > 



1 Tl' 6 

eT2;rFT < <e^. (16) 

V27m n" 

In the following lemma we are only interested in the lower bound on Bp(k\n). 
However, the upper bound comes for free. 

Lemma 8. For < k < n and p G [0, 1] 



Proof. We get 



Sp(fc|n)y&!i^2"^(^llf) 

r 

fc!(n- fc)! 



^ (n — /^V V 



2(k Iog(p) + (Ti-fc) log(l-p)) 



A:!(n-fc)! 

/ k{n-k) n\ /■k\'' /n-Zcx"-*^ 

V » ' A:!(n-fc)! ' VnJ ' V n J 



n!e" V2^k'' ^27r(n - A:) (n - fc) 



Using (|16|l three times we obtain 



— (17) 



Analogously (and since either ^ < or ^ < i^^J^^J 



Bp(fc|n) J27rM!iz±)2"-D(^llp) < eT5^e-™e-T2(Tr^:fe)+T < 1. □ 
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Corollary 1. For p £ [5,1], and pn < k < n 



Proof. From Lemma |S1 we get 



\l2TTk{n-k) 

where we used k > ^. Using the esthxiate in Lemma [7| concludes the proof. □ 
Lemma 9. Let p £ [i, 1], n, s e N such that pn + 3s < n. Then, 

k— \pri \ +s 



S 

■ g Tip(l-p) _ 



Proof. Clearly, ^^J^^^ > ^, for all values of n and k in the sum. Since k < 

pn + 2s for all values in the above sum we get ^— P < and also we see that 
n — k > s. Using this together with Corollary ^ thus implies for all k of interest 

^ — r) ^ — 2i 



BJk\n)>e^^\ — e 
V 7rn 

1 



Since there are s summands we get the lemma. □ 
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